Skip to content
This repository has been archived by the owner on Jan 21, 2021. It is now read-only.

Create New SSZ Factory for Arrays of Roots #91

Merged
merged 31 commits into from
Oct 21, 2019
Merged

Conversation

rauljordan
Copy link
Contributor

@rauljordan rauljordan commented Oct 18, 2019

No tracking issue.

Background

This PR works on optimizing one of our biggest bottlenecks in Prysm, which is the usage of hash tree root on basic arrays of roots. If a single element changes, we have to recompute the entire function again, making the processing of slots in the beacon state extremely slow when called repeatedly.

Changes

This PR introduces the following changes:

  • A new ssz interface implementation for arrays of [32]byte roots with special caching mechanisms
  • A ToggleCache public function that allows for disabling the cache when running ssz related items in spec tests for preventing collisions
  • Adds some benchmarks for ssz hash tree root of arrays of roots with and without caching

@rauljordan rauljordan self-assigned this Oct 18, 2019
@rauljordan rauljordan added the Tracking For tracking changes label Oct 18, 2019
@codecov
Copy link

codecov bot commented Oct 20, 2019

Codecov Report

Merging #91 into master will decrease coverage by 3.6%.
The diff coverage is 55.86%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #91      +/-   ##
==========================================
- Coverage   78.99%   75.39%   -3.61%     
==========================================
  Files          12       13       +1     
  Lines        1157     1272     +115     
==========================================
+ Hits          914      959      +45     
- Misses        138      190      +52     
- Partials      105      123      +18
Impacted Files Coverage Δ
ssz.go 84% <100%> (ø) ⬆️
types/struct.go 77.72% <100%> (ø) ⬆️
types/string.go 77.14% <100%> (ø) ⬆️
types/determine_size.go 85.15% <100%> (+0.6%) ⬆️
types/slice_composite.go 70.83% <100%> (ø) ⬆️
types/basic.go 83.72% <100%> (ø) ⬆️
types/slice_basic.go 79.13% <100%> (ø) ⬆️
types/array_composite.go 65.88% <100%> (ø) ⬆️
types/factory.go 100% <100%> (ø) ⬆️
types/array_basic.go 57.74% <42.85%> (-17.26%) ⬇️
... and 3 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 90e28a2...f39ed31. Read the comment docs.

@rauljordan rauljordan marked this pull request as ready for review October 20, 2019 04:06
@rauljordan rauljordan changed the title Create New SSZ Factory for Arrays of Roots WIP: Create New SSZ Factory for Arrays of Roots Oct 20, 2019
Copy link
Contributor

@0xKiwi 0xKiwi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks absolutely amazing, great work!!

ssz_test.go Outdated Show resolved Hide resolved
@rauljordan rauljordan added the ready for review Ready for review label Oct 21, 2019
@rauljordan rauljordan changed the title WIP: Create New SSZ Factory for Arrays of Roots Create New SSZ Factory for Arrays of Roots Oct 21, 2019
@@ -25,31 +25,18 @@ func newBasicArraySSZ() *basicArraySSZ {
}
}

func (b *basicArraySSZ) Root(val reflect.Value, typ reflect.Type, maxCapacity uint64) ([32]byte, error) {
func (b *basicArraySSZ) Root(val reflect.Value, typ reflect.Type, fieldName string, maxCapacity uint64) ([32]byte, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do you need fieldName ? it isnt used in basicArraySSZ

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to add it everywhere because all of these functions need to meet the same interface, and it could be useful down the line for further caching or namespacing

var enableCache = false

// ToggleCache enables caching of ssz hash tree root. It is disabled by default.
func ToggleCache(val bool) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is only for testing ,correct ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm to some extent - it is nice to have the functionality of disabling caching though, either for benchmark reasons or for other reasons. It is the best compromise I found to prevent collision resistance in tests and maintain speed.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds good. Would be an easy way to debug the cache if we have anything wrong

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
ready for review Ready for review Tracking For tracking changes
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants